Journal of the American Medical Informatics Association

Potentially inappropriate medications expose older adults to preventable harm, yet deprescribing remains difficult to implement consistently. Although electronic health record (EHR) interventions can reduce prescribing, health systems lack clear evidence about which routinely captured patient, primary care provider (PCP), and intervention-design factors predict medication discontinuation or dose tapering. Understanding these determinants is essential for targeting and scaling deprescribing support. In this study, we conducted the first machine-learning analysis of these trial data. We analyzed 2,979 adults aged 65 years or older and 158 structured EHR features spanning patient characteristics, PCP characteristics and EHR-use behaviors, and deprescribing-tool design. We compared eight models for predicting medication discontinuation or dose tapering and used SHAP to examine feature importance. TabPFN achieved the highest positive predictive value (67.71%), AUROC (74.30%), and AUPRC (60.95%), although overall predictability was moderate. Our findings show that even rich structured EHR data only moderately predict deprescribing, suggesting that important clinical determinants are not captured in routine fields. PCP EHR-use measures accounted for 19 of the 25 highest-ranked TabPFN features, although they also constituted most candidate predictors. The study provides the health system with an informative reference for predicting high-risk medication deprescribing. Future models should incorporate richer clinical context and undergo external validation before informing personalized deprescribing support.

Objective: To develop and evaluate machine learning methods for automated identification of family relationships from electronic health records (EHRs). We employed Random Forest classifiers to identify five relationship types: Mother-Child, Father-Child, Sibling-Sibling, Twin-Twin, and Partner-Partner, using a four-stage iterative refinement process to improve precision for patient-centered outcomes research (PCOR) and healthcare applications. Materials and Methods: We used two large-scale Indiana datasets: the Indiana Network for Patient Care (INPC), comprising approximately 15 million unique individuals across approximately 45 million medical records, and the Indiana Natality dataset (birth certificates, 1970-2024), which served as the gold standard. Positive cases were derived by linking verified relationships from Natality records to INPC using a shared Global Identifier. Negative cases were drawn from a three-tier blocking strategy combined with sliding window restriction and similarity scoring, which together reduced the comparison space from over 1014 potential pairs to approximately 121 million candidates. Five Random Forest classifiers underwent four- stage iterative refinement: initial training, feature removal for multicollinearity, feature engineering to address systematic errors, and training-data refinement restricted to relationship pairs with at least one shared contact feature. Results: The models achieved precision of 0.92 to 1.00, recall of 0.97 to 1.00, and F1 scores of 0.94 to 1.00 across all relationship types (Mother-Child 0.97, Father-Child 0.98, Sibling-Sibling 0.98, Twin-Twin 1.00, Partner-Partner 0.94). Between 15% and 78% of Natality-verified relationships lacked any shared contact information in INPC and were structurally undetectable by record linkage; restricting training and evaluation to linkable pairs improved precision, and training-data refinement improved F1-scores by a further 0.02-0.04 for the Mother-Child, Father-Child, and Sibling models. High-confidence predictions (probability [≥] 0.9) captured 77-99% of true positives. Age difference was the primary predictor for parent- child and twin relationships, while phone number similarity was most important for siblings. Discussion: The framework identifies family relationships from standard demographic fields available in most EHR systems. The iterative refinement approach shows how systematic error analysis can guide 1 methodological improvements. Training-data refinement addresses a constraint in EHR-based family linkage: not all verified biological relationships have overlapping demographic footprints in healthcare data. Unlike rule-based approaches, the framework provides probability scores that enable configurable deployment thresholds. Conclusion: The Random Forest models demonstrate high performance suitable for large-scale research deployment and are ready for testing in clinical applications rather than immediate widespread clinical use. The methodology can be adapted to other EHR systems, supporting family-centered study design in patient- centered outcomes research. Keywords: family linkage, electronic health records, machine learning, Random Forest, record linkage, patient-centered outcomes research, health informatics

Large language models (LLMs) screen titles and abstracts without review-specific training, but generating screening decisions as text takes processing time and incurs API charges. We evaluated Jev, a non-generative model returning classification probabilities, on 4527 records from two systematic reviews of bipolar disorder treatments. We used human reviewers' decisions to retain records for full-text assessment as the reference standard. In the primary analysis, we asked Jev yes-or-no questions ("Noul") about whether to retain each record. Records were retained when their retention probability was at least 50%, a cutoff specified before the final evaluation runs. We also evaluated lower retention thresholds. For comparison, we gave Jev explicit include/exclude options ("Choice") and used the same prompts with GPT-5 mini and GPT-6 Astra; both LLMs used default reasoning. In the primary analysis at 50%, sensitivity and specificity were 99.6% and 98.9% for light therapy and 93.2% and 93.5% for adjunctive pharmacotherapy. Lower cutoffs selected on these data retained all reference-positive records. Compared with assessing every record manually, these cutoffs reduced the number requiring human assessment by 96.9% for light therapy and 61.3% for adjunctive pharmacotherapy. For light therapy and adjunctive pharmacotherapy, sensitivity was 92.3% and 90.0% with Jev using explicit options at 50%, 87.2% and 90.0% with GPT-5 mini, and 92.3% and 80.8% with GPT-6 Astra. Costs for screening all 4527 records once were US$0.20 for Jev using explicit options, US$2.59 for GPT-5 mini, and US$33.45 for GPT-6 Astra. Jev's probabilities could provide graded information for reviewers assessing titles and abstracts.

A safety certificate for a clinical AI promises that no more than a stated fraction of the answers it releases carry a defect that would change care. That promise is computed from past cases a clinician marked safe, and clinicians miss things. A missed defect is not filed as a disagreement, it is filed as a safe case, so the certificate rests on a record cleaner than the ward ever was and has to be widened to compensate. Widening it correctly takes one number: how often that clinician misses a defect the model produced. We show that number cannot come from the only test a hospital can run on a working reviewer, planting a known defect and counting the catches. A planted defect is as hard to spot as whoever planted it made it, a real defect is as hard to spot as the model happened to make it, and comparing the two needs the very number being sought. No safety factor recovers it, because one wide enough for every miss rate releases nothing. The failure also runs the dangerous way, because corrections that turn noise into one rate need the mistakes to be unrelated to the case, whereas a defect is missed because it read well, so the risk on record sits below the true risk at every threshold. Across four simulated reviewers that condition never holds, and testing it on a separately drawn sample is affordable only where the answer was never in doubt: a reviewer missing 85% of defects costs 185 adjudicated cases against 147,938 for one missing 3%. Adjudicating the reviewer's own cases instead removes that test and most of that cost, 498 cases for the careful reviewer, so modelling the noise costs more than measuring it. Our library therefore refuses the widened certificate unless the condition is confirmed on adjudicated data, and fails closed otherwise.

Electronic health record (EHR) data vary substantially in documentation density across patients, independent of disease burden. Existing tools such as the Charlson Comorbidity Index (CCI) and Elixhauser Comorbidity Index measure disease burden but do not capture differences in data volume, leaving a common source of bias unaddressed in EHR-based analyses. To address this gap, we developed the EHR Density Index (EDI), which characterizes the quantity, depth, and breadth of EHR data per patient per year, normalized by utilization patterns, using records from 24,987 adult patients at UNC Health (2018 - 2024). The EDI combines a utilization cluster assigned via Gaussian Mixture Model with within-cluster residuals quantifying documentation volume across four clinical domains. Four interpretable clusters emerged; while CCI predicted cluster membership, its associations with within-cluster residuals were weak, confirming the EDI captures dimensions of the patient record distinct from disease burden. The EDI is intended as a covariate to address documentation density as a source of confounding in real-world data-driven research.

Objective: To compare the accuracy, cost, and speed of a low-cost, non-generative structured-judgment model and four general-purpose large language models (LLMs) for extracting smoking status, pack-years, and quit date from clinical notes to support lung cancer screening (LCS) clinical decision support (CDS). Materials and Methods: We built a synthetic, shareable benchmark of 3,000 outpatient notes in three conditions: 1,000 template-generated notes, 1,000 realistic, ``messy'' notes written by an LLM from structured facts, and 1,000 ``messy'' notes that required complex arithmetic to determine pack-years and quit dates. Ground truth reference labels were programmatically generated before each note was created. We compared TypeSafe Jev 1.13 with Claude Haiku 4.5, Claude Sonnet 5, GPT-6 Luna and GPT-6 Sol, each using an identical structured output schema. We determined United States Preventive Services Task Force (USPSTF) 2021 and American Cancer Society (ACS) 2023 eligibility from each system's output in code. Results: Jev made the correct eligibility decision for 99.1%, 99.9%, and 94.4% of notes in the three conditions, compared with 97.8%, 98.4%, and 98.1% for Haiku, 100.0%, 99.6%, and 99.8% for Sonnet, 99.7%, 98.8%, and 98.5% for Luna, and 100.0%, 99.8%, and 99.6% for Sol. In the complex condition, Jev produced 27 false positive and 14 false negative screening flags per 1,000 notes, the most of any system. Luna cost $0.12 to $0.21 per 1,000 notes and Jev $0.61 to $0.64, compared with $2.44 to $4.22 for Sol, $3.37 to $4.43 for Haiku, and $3.76 to $6.61 for Sonnet. Jev was the fastest system (median 0.45 to 1.21 seconds per note, compared with 1.71 to 3.50 seconds for the others). Discussion: All models tested had high accuracy. The structured-judgment model had the lowest latency, which could allow for synchronous use in clinical decision support systems rather than batched, cached invocation. The cost of the structured-judgment model was low, but the newest cost-efficient LLM, Luna, was less expensive and more accurate on the complex notes, though its latency was higher. Accuracy, latency, and cost are all moving quickly as new models are released. Conclusion: Fast, low-cost language models could make real-time, note-based CDS practical, and current models are highly accurate. Prices and capabilities are changing rapidly, so continuous benchmarking and system updates are essential.

Objective. Privacy regulation constrains sharing line-level electronic health records (EHR) across institutions. One alternative is to aggregate counts into a cube, a table of counts for every combination of categorical variables, with cells below a threshold suppressed. This study asked whether common analyses on the cube reproduce conclusions from line-level data, and whether suppression prevents recovery of the small cells it is meant to hide. Materials and Methods. A Bayesian count-inference pipeline was built that reconstructs suppressed counts and doubles as a reconstruction attack. Applied to 285 pediatric kidney-transplant patients at Boston Children's Hospital, statistical fidelity (Jensen-Shannon divergence, Cramer's V, and R2) and analytical utility (marginal distributions, subgroup graft rejection odds ratios, and logistic-regression classification) were evaluated. Conditional Tabular GAN (CTGAN) synthetic data served as a comparator. Results. Statistical analyses on the cube recapitulated results from line-level data. Across 106 demographic-by-medication subgroups, a bootstrap mean of 3.5 subgroups showed a significant graft-rejection association. The cube's odds-ratio sign changes reversed no significant associations, versus 2.3 for CTGAN. The same reconstruction also defeated suppression: in a 10-variable cube, 76.6% of suppressed cube cells were recovered exactly (14,554 of 18,994), including 85.5% of single-patient cells. Discussion. The cube reproduced common kidney-transplant analyses, but the same reconstruction also recovered suppressed cells; fidelity and privacy risk are thus two faces of one reconstruction rather than independent properties. Conclusions. The cube is a useful surrogate for these kidney-transplant analyses only when paired with a stronger privacy mechanism. This study demonstrated reconstructability of suppressed counts, not re-identification.

Title and abstract screening is among the most labor-intensive steps of systematic reviews. Jev, a low-cost AI decision model, scores records against eligibility criteria without screening labels. We scored 169,982 records from 128 reviews and, after tuning on development reviews, evaluated 23 held-out reviews once and 31 Cochrane reviews post hoc. Jev outranked the best free ranker in all 23 held-out reviews (mean AUC 0.966 vs 0.831) and came within 0.011 AUC of Claude Opus 5.5 on 1999 records at about 1/700 of its list-price cost (US$0.02/1000 records). Combined with active learning, it raised work saved at 95% recall from 0.659 to 0.806. A fixed score threshold fell below 95% recall in 3 of 28 Cochrane reviews, whereas Jev ranking with statistical stopping (post hoc) reached 95% recall in all 51 reviews, reading 86-88% of records. Label-free ranking with Jev was accurate and inexpensive, but stopping safely required screening decisions.

Symbolic clinical natural language processing (NLP) systems remain widely used for extracting clinical concepts from electronic health record (EHR) narratives, but maintaining rule resources requires extensive manual error analysis and rule refinement. This study investigates whether large language models (LLMs) can assist in identifying extraction errors and generating candidate rules to improve symbolic clinical NLP systems. Using error reports derived from a multi-site evaluation of a previously validated symbolic model for cognitive and neuropsychiatric-related clinical concepts, we developed a human-in-the-loop framework, REFINE. The framework first uses LLMs to classify extraction errors and generate explanatory reasoning, which can then be incorporated into prompts for rule generation. Three LLMs (GPT-5.2, GPT-4o, GPT-4o-mini) were evaluated under four prompting conditions. LLM-generated rule sets improved performance compared with the baseline NLP-CAM system, increasing F1-score from 0.37 to 0.58. These findings suggest that LLMs can support scalable rule refinement for symbolic clinical NLP systems.

Medical systematic reviews are central to evidence-based medicine, but they remain slow, labor-intensive, and difficult to maintain under the full Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) workflow. Recent LLM-based deep research agents offer a promising route to addressing this challenge, yet reliable deployment in medical systematic reviews remains limited by insufficient clinical domain knowledge and inconsistent adherence to evidence-based methodological standards across the full workflow. We address these gaps with MedSR-Copilot, a PRISMA-aligned multi-agent copilot that decomposes review automation into literature retrieval, coarse-to-fine screening, data extraction, Risk-of-Bias assessment, and evidence synthesis, while preserving structured intermediate artifacts throughout the workflow. We further introduce MedSR-Bench, an end-to-end benchmark for evaluating systems beyond isolated subtasks, from review input to final evidence-synthesis conclusions. MedSR-Copilot completes medical systematic reviews end-to-end under the full PRISMA workflow, achieving 63.6% human-aligned conclusions, 18.3 percentage points above the best baseline among strong general-purpose LLMs and prior automated review systems. In a human-AI collaboration study involving 23 analysis groups across four systematic review topics, MedSR-Copilot, used as a copilot, reduces end-to-end review time by 64.9% and improves final conclusion accuracy by 27.4 percentage points compared with routine-practice workflows. Together, these results demonstrate the reliability and efficiency of MedSR-Copilot as a medical research copilot and suggest a practical path toward trustworthy review automation.

Background. Chest radiographs (CXRs) are the most common imaging exam performed, but CXR reports can be nonspecific due to incomplete history, relying on vague terms like opacities to convey diagnostic uncertainty. We prospectively evaluated whether introducing a clinical dashboard with relevant patient information affects CXR report specificity and efficiency, report concordance with discharge diagnoses, and follow-up chest CT utilization. Methods. In this stepped-wedge cluster randomized trial (NCT06785246), a clinical dashboard for radiologists was rolled out with user education sessions across 15 service areas of an integrated health system from June 2024 to June 2025. Emergency department (ED) and inpatient CXRs were included. A large language model (GPT-4o) was validated using a Delphi panel of three experts to classify report specificity and CXR report diagnosis. A subsequent model release (GPT-5.4) graded each CXR report impression as negative, specific abnormal, or nonspecific abnormal. It also judged whether an abnormal impression matched an ED discharge diagnosis. The primary outcome was the proportion of specific reports (negative or specific abnormal). Secondary outcomes were report-discharge diagnosis concordance, follow-up chest CT, antibiotic and diuretic orders, and CXR reporting time. Mixed-effects models adjusted for time, patient age, and radiograph view, with random effects for radiologists, facilities, and service area. Results. The trial included 404,860 ED and inpatient encounter STAT priority CXRs read by 387 radiologists (256,303 control; 148,557 intervention). The adjusted proportion of specific reports was 82.2% in control periods and 82.1% in intervention periods (difference, -0.10 percentage points; 95% CI, -0.53 to 0.34; P=.66). Among abnormal reports, concordance with a discharge diagnosis rose from 40.9% to 42.2% (difference, 1.24 points; 95% CI, 0.20 to 2.28; P=.02). Follow-up chest CT fell from 11.4% to 10.9% (difference, -0.55 points; 95% CI, -0.92 to -0.17; P=.004). Antibiotic orders (24.0% vs 23.9%) and diuretic orders (14.7% vs 14.8%) within 7 days did not change, nor did reporting time (difference, -0.8 seconds; 95% CI, -2.7 to 1.0). Conclusions. Giving radiologists EHR context through a dashboard did not change the specificity of STAT priority CXR reports or prescribing patterns. It was associated with small improvements in diagnostic concordance and follow-up CT use, without slowing reporting. Validated large language models make it feasible to measure report quality across a health system.

Objective: To estimate Long COVID undercoding in electronic health records (EHRs) against federal survey prevalence, and to evaluate a predictor of later coding that anyone can rebuild from counts a research enclave releases. Materials and Methods: Among 23,790 All of Us adults with COVID-19, we compared coded shares with Household Pulse Survey prevalence. Among 23,300 adults uncoded at day 90, we predicted a first U09.9 code by day 455. We fitted naive Bayes and linear discriminant models outside the enclave from released counts and compared them with an elastic net fitted inside to the same patients and features. Tests were a random 20% of patients (4,680, 50 coded) and 18 held-out EHR sites (5,760, 60 coded). Results: Under survey-comparability assumptions, about 1 in 9 adults who ever had Long COVID carried its code, and the women-to-men coding ratio was 0.79 (95% sensitivity interval, 0.65-0.95). The released models' areas under the receiver operating characteristic curve (AUROCs) were 0.68 (95% CI, 0.60-0.75) on the patient test and 0.64 (0.55-0.72) at held-out sites, 0.049 (0.001-0.104) above the elastic net. On the patient test, the elastic net had a higher area under the precision-recall curve (difference 0.062, 0.003-0.122). Released probabilities overestimated coding risk at held-out sites. Discussion: Fitting from released counts did not consistently lower discrimination. Coded outcomes inform disease discrimination only under stated coding assumptions. Conclusion: Long COVID is substantially and unevenly undercoded, and a model rebuilt from public counts predicts its later coding with moderate discrimination.

Clinicians routinely make decisions for patients whose comorbidities, prior therapies, laboratory abnormalities, or demographic characteristics are not reflected in the supporting evidence, even though these factors can influence treatment selection. Given this gap in the evidence base, we sought to evaluate how AI models respond to clinical questions in which patient characteristics or the treatment comparison and outcome of interest may affect the applicability of available evidence. We developed and publicly released Precision Evidence Bench, a benchmark of 209 synthetic clinical questions with specified population, intervention, comparator, and outcome (PICO) elements. Each answer is assessed using a five item rubric that evaluates whether a single cited source matches the PICO and whether the answer faithfully reports its sources. The item verdicts determine one of five ratings, ranging from Precision match to Unsupported claim, without requiring a reference answer. We evaluated four frontier large language models (LLMs) using web search across all 209 questions, then repeated the evaluation with additional access to a corpus of precision evidence-based findings (pEBFs) designed to address these evidence gaps, using two corpus sizes. For every model, Precision match rates increased with the amount of precision evidence available, from 13 to 18% with web search alone to 17 to 28% with 100 million pEBFs and 41 to 56% with 500 million pEBFs. These findings suggest that access to matching evidence is an important determinant of LLM performance on precision medicine questions.

Background: Clinical terminology pipelines must first extract candidate spans from narrative notes and then determine whether those spans map to existing concepts or warrant further review. Evaluation is difficult because span boundaries vary between annotators and because downstream decisions depend on the terminology evidence retrieved for each span. Objective: We evaluated clinical concept extraction, terminology linking across controlled evidence conditions, and ontology-extension triage for terms that remained unmatched after initial terminology screening. Methods: We conducted 3 complementary pilot evaluations that used distinct units of analysis and were analyzed separately. Study 1 compared 5 automated extraction pipelines and a union-merge analysis with 2 human annotation sets in 66 deidentified clinical notes from 3 health systems. Agreement was evaluated by exact string matching and BGE-large-en-v1.5 embedding matching. Study 2 evaluated 56 clinical spans, including 28 with reference Unified Medical Language System concepts and 28 adjudicated as unsuitable for ontology extension, under complete retrieval, matched-concept masking, and large language model-only inference, yielding 168 span-condition outputs. The graph retrieval pipeline used BGE-large-en-v1.5 embeddings, and the decision model was Gemma 3 27B. Study 3 applied full vector retrieval to 84 terms previously not matched in either UMLS or BioPortal. Results: In Study 1, interannotator exact-match F1 was 0.29 and embedding-match F1 was 0.75. Automated exact-match F1 scores ranged from 0.07 to 0.17; embedding-match F1 was highest for MedGemma (0.55), followed by Gemma (0.53), sci_md and SciBERT (each 0.43), and Llama 3.3 (0.32). In Study 2, complete retrieval returned a reference-matched link for 28/28 known-concept spans (100%; 95% CI, 87.9%-100%). Masking assigned POSSIBLE_CANDIDATES to all 28; large language model-only inference assigned POSSIBLE_CANDIDATES to 25/28 (89.3%) and LINKED to 3/28 (10.7%). Across the 3 evidence conditions, the same 12/28 unsuitable-extension spans were classified as NOT_MEANINGFUL (42.9%) and the same 16/28 as POSSIBLE_CANDIDATES (57.1%). In Study 3, the pipeline assigned PLAUSIBLE_EXISTING_CONCEPT to all 84 terms, none was flagged for extension, and top-candidate similarity averaged 0.914 (SD 0.027); extension status was not independently adjudicated. Conclusions: Measured extraction performance varied substantially by matching definition, whereas exact-link decisions varied with the availability of matched terminology evidence. In the follow-up sample, initial nonmatching did not establish ontology novelty: after semantic retrieval, the pipeline classified all 84 terms as plausible existing concepts and proposed none for extension. These findings support separate evaluation of extraction, retrieval, evidence-grounded linking, and extension candidacy.

Large language models (LLMs) are increasingly used to draft medical manuscripts, yet their citations are unreliable and clinicians lack a validated way to verify them. We evaluated three frontier LLMs, Claude Opus 4.8, GPT-5.5, and Gemini 3.5 Flash, generating 270 cardiology narrative reviews with web search enabled, and verified all 8,050 references against PubMed. Problematic references accounted for 11.5% of GPT-5.5 output, 29.2% of Claude output, and 29.6% of Gemini output (P < 0.001), with no significant gradient across topics of differing publication volume (P = 0.052). Misattribution, a valid PubMed identifier that resolves to a different article, made up 77% of errors, whereas fabrication was rare (0.6%). Against an expert-adjudicated set of 270 references, an LLM-based Chain-of-Verification (CoVe) detected 60 of 62 problematic references (sensitivity 96.8%, specificity 98.6%), including every misattribution and fabrication. LLM-generated citations require identifier-level verification, and CoVe provides it at expert-level accuracy.

Background: Free-text notes in electronic health records (EHRs) contain fine-grained psychiatric information that is essential for psychiatric research and clinical care, and often absent or under-recorded in structured codes alone. Clinical natural language processing (cNLP) can support extraction of this information from EHR notes, yet Spanish-language cNLP remains under-developed. Moreover, broad evaluations comparing multiple encoder-based language models across extensive, fine-grained psychiatric concept sets remain scarce, and it remains unclear how these models compare with traditional NLP (tNLP) systems and much larger generative large language models (LLMs). In addition, cross-site performance of fine-tuned models is rarely tested, and limited annotated training data remains a major challenge, especially for rare symptoms. Objectives: We aimed to advance scalable, global psychiatric cNLP by fine-tuning multiple encoder-based models with differing architectures and pre-training strategies for detecting fine-grained psychiatric concepts in Spanish EHRs. We further evaluated the impact of augmenting the fine-tuning data with precision-weighted weak labels for less-frequent concepts, and compared the performance of the encoder-based models to that of tNLP and a fine-tuned generative LLM trained on the same data. Finally, we evaluated model cross-site generalizability on an external EHR dataset. Methods: Three encoder-based models (BETO, XLM-RoBERTa-large, and bsc-bio-ehr-es) were fine-tuned on 1,642 clinician-annotated EHR documents from Colombia to detect 110 psychiatric concepts in Spanish text. To address the limited annotated examples available for less-frequent concepts, 12,000 additional documents were weakly-labeled for less-frequent concepts using tNLP, and incorporated into the fine-tuning data with labels weighted by pattern precision. Models were compared with tNLP and a generative LLM, and evaluated on an external EHR dataset from another psychiatric hospital in Colombia. Results: Encoder model performance varied substantially, with macro-F1 ranging from 0.64 to 0.81. BETO achieved the highest macro-F1 (0.81; median F1=0.88 [IQR=0.77-0.96]). Adding precision-weighted weak labels for less-frequent concepts improved BETO's overall macro-F1 to 0.83 and increased mean F1 for the 55 augmented concepts from 0.82 to 0.86. Under matched fine-tuning conditions, fine-tuned BETO and the tNLP method were equivalent in F1, whereas the LLM significantly outperformed BETO in F1. After weak-label augmentation, BETO significantly outperformed tNLP in F1 (PFDR<.001) and narrowed the performance gap with the LLM, although equivalence was not established. Lastly, fine-tuned BETO maintained reasonably strong performance on data from an external hospital not used for model fine-tuning (out-of-domain macro-F1=0.78). Conclusions: General-purpose pre-trained encoders had strong performance for psychiatric concept extraction from Spanish EHRs. Weak-label augmentation improved BETO's performance and strengthened results relative to a tNLP baseline, while reducing, but not eliminating, the performance gap with a much larger fine-tuned generative LLM. These findings highlight the utility of these relatively lightweight models for scalable, accurate and reproducible detection of psychiatric concepts in Spanish-language EHRs.

Importance: The Medicare Physician Fee Schedule values ambulatory services using survey-based estimates of physician time, a costly process vulnerable to inaccuracy. EHR audit logs are passively and universally collected and likely meet the American Medical Association Relative Value Scale Update Committee (AMA RUC) Extant Database Standards for alternative data to inform service valuation estimates. However, audit logs' validity for estimating face-to-face time, a core component of many of the most used procedural codes, is unknown. Objective: To assess whether EHR audit logs can facilitate accurate estimation of physician-patient face-to-face time during ambulatory visits. Design: Single-site validation study pairing EHR audit logs with direct audio/video recordings of ambulatory primary care visits (reference standard). We assessed agreement between audit-log estimates and recorded visit duration with Bland-Altman analysis and mean absolute error. Setting: University of Pennsylvania Observer data repository, a multi-modal database of ambulatory visits to Penn Medicine primary care clinics including video and audio recordings, transcripts, clinical EHR data, and EHR audit-log data. Participants: 75 visits from 12 physicians with data collected between August 2023 and February 2025. Exposures: A rule-based algorithm estimated face-to-face time via clusters of physician EHR activity relative to patient check-in time. Inactivity thresholds were applied to differentiate intraservice time with no EHR activity (e.g., a physical exam) from the end of a visit, with longer thresholds used for longer visits. Main Outcomes and Measures: Duration of recorded face-to-face visit time. Results: Recorded face-to-face time averaged 31.6 minutes (SD, 13.9). The audit-log estimator was unbiased (mean bias: -0.63 minutes; 95% CI: -6.0 to 2.6; mean absolute error: 9.0 minutes) and estimated face-to-face time within approximately 2% (38 seconds) of the recorded visit time with no proportional bias (slope: 0.13; P = .25). The 95% limits of agreement were -25.7 to 24.4 minutes, and 37% of estimates fell within 10% of recorded duration. Conclusions and Relevance: In this validation study, EHR audit logs recovered unbiased estimates of ambulatory face-to-face time. Larger-scale studies are required to verify generalizability, but EHR audit logs offer a scalable means to improve the rigor of retrospective physician work relative value unit estimation. This study represents a first step in a potential paradigm shift for measuring components of the fee schedule across procedures, settings, and provider types.

Objective Translating free-text clinical trial criteria into computable code sets is a valuable standardization practice that is necessary for producing reproducible real-world evidence studies but requires standardized interpretation across multiple clinical vocabularies. Methods We developed TrialCode Agent, a hybrid-large language model (LLM)-terminology verification agent that generates, formats, verifies, and expands candidate codes from free-text clinical criteria. The system supports ICD-9-CM diagnoses and procedures, ICD-10-CM, ICD-10-PCS, LOINC, and RxNorm medication concepts. We compared Baseline, Hybrid biomedical retrieval-augmented generation (RAG), and terminology-guided Family expansion pipelines using Claude, GPT Qwen, and MedGemma on 40 criteria from 11 trial groups. Performance was evaluated against expert-built reference code sets using exact-code precision, recall, and F1. Results The optimal pipeline varied by model. Claude with Baseline achieved the highest performance (precision 0.755, recall 0.619, F1 0.680), followed by GPT-5.5 with Baseline (precision 0.569, recall 0.658, F1 0.610), Qwen with Hybrid biomedical RAG (precision 0.656, recall 0.470, F1 0.548), and MedGemma with Family expansion (precision 0.487, recall 0.316, F1 0.383). Hybrid biomedical RAG improved aggregate F1 only for Qwen but increased GPT-5.5 RxNorm F1 from 0.320 to 0.909. Macro-averaged results showed criterion-level gains despite lower micro-averaged aggregate performance. Family expansion increased recall across models but generally reduced precision. In staged verifier ablation, micro-F1 increased from 0.254 before verification to 0.505 after final verification and expansion. Existence/vocabulary checking removed 2,594 false-positive codes, and acceptance filtering removed 952 additional false-positive codes before controlled expansion. Conclusions Combining LLM-based clinical interpretation with deterministic terminology verification produces auditable, database-ready code sets, but retrieval and broad family expansion do not consistently improve exact-code performance. Retrieval was particularly useful for RxNorm mapping, whereas overly broad or incomplete candidate generation remained the main source of error. Deterministic verification improves code validity and query readiness but cannot replace accurate clinical interpretation.

Background. Health systems answer most questions by having expert analysts hand-write queries against a complex electronic health record data warehouse, a slow, resource-intensive process. Whether an autonomous coding agent can do this accurately is unknown. Methods. In a single-center quality-improvement evaluation, we posed ten questions about a common pediatric infection, acute otitis media. The questions went to analysts, whose adjudicated answers were the reference, and to an autonomous coding agent (OpenAI Codex), which wrote and ran read-only queries on a full copy of the production data warehouse (Epic Caboodle). We ran the agent under four conditions: an autonomous baseline (each question answered three times at two reasoning-effort settings), a variant in which it listed its assumptions, interactive analyst feedback, and reuse of a corrected definition across related questions. Outcomes were accuracy, patient-level agreement (F1), reproducibility, and cost. Results. Working autonomously, the agent wrote valid queries and never fabricated data, but rarely produced the exact answer. At medium effort, it came within 5% of the reference on 27 of 30 runs but exact on only 10. Higher effort produced no improvement. Reproducibility was the greater weakness, with the three runs returning an identical answer on only 3 of 10 questions. When a run matched the reference, it had found the same patients (F1 = 1.00); the exception was an over-counted procedure (F1 = 0.72). With analyst feedback on four questions, it answered two exactly, and reusing the corrected definition on related questions restored reproducibility and accuracy. Conclusions. A general-purpose coding agent matched an adjudicated analyst reference on most routine questions but not reproducibly, because key definitions depended on warehouse knowledge that the data dictionary omits. Supplying that knowledge, by prompt or feedback, restored reproducibility. Under expert analyst supervision, the agent is already a capable drafting aid and a promising step toward broader hospital analytic support.

Background Large language models (LLMs) show promise in scientific literature reviews, but cloud-based proprietary models like GPT-4 raise concerns around privacy, copyright, and licensed article access. Using a scoping review on AI applications for treatment effect estimation as a use case, this study evaluated local open-source LLMs for automated article screening in scientific literature reviews and developed a practical workflow for their implementation. Methods We extracted a sample dataset of 300 records from an ongoing scoping review on machine learning methods for treatment effect estimation using real-world medical data. We divided the dataset into training (n = 200) and testing (n = 100) datasets. We evaluated 11 locally deployed open-source LLMs from five model families (GPT-OSS, Gemma, Qwen, Mistral, Llama) for both abstract screening and full-text screening. Full-text screening was done using text converted by two PDF-to-text conversion tools: Docling and PyMuPDF4LLM. Performance was assessed using raw accuracy with 95% Wilson's confidence interval, relaxed accuracy, Cohen's kappa, and runtime. During workflow development, prompt engineering, output extraction strategies, and generation settings were iteratively optimized using the training dataset. Results GPT-OSS-20B achieved the highest raw accuracy for title and abstract screening (0.890 on the training set and 0.850 on the test set), followed closely by Gemma-4-31B-it (0.875 on the training set and 0.830 on the test set) and Gemma-4-26B-A4B-it (0.810 on the training set and 0.790 on the test set). In full-text screening, slightly higher raw accuracy and reduced false-positive rates were observed, with GPT-OSS-20B and Gemma-4 variants consistently demonstrating strong performance. Only minor variations were found between the two PDF-to-text conversion methods evaluated, though runtime varied substantially across models. Based on these findings, we compiled a set of practical recommendations for local LLM-assisted article screening. Conclusions Local LLMs can effectively support literature screening when integrated into a carefully designed workflow. Recent generations of open-source local models achieved performance comparable to human reviewers and proprietary LLMs. Local deployment also provides advantages in data privacy, copyright compliance, customization, and cost-efficiency. While human oversight remains essential, the proposed workflow demonstrated that local LLMs represent a practical and reproducible approach for accelerating evidence synthesis.